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Abstract: Building an evidence base for healthcare interventions has long been advocated as both professionally and 
ethically desirable. By supporting meaningful comparison amongst different approaches, a good evidence base has been 
viewed as an important element in optimising clinical decision-making and the safety and quality of care. Unsurprisingly, 
medical research has put considerable effort into supporting the development of this evidence base, and the randomised 
controlled trial has become the dominant methodology. Recently however, a body of research has begun to question, not 
just this methodology per se, but also the extent to which the evidence it produces may marginalise individual patient 
experiences, priorities and perceptions. 

Simultaneously, the widespread adoption and utilisation of information systems (IS) in health care has also prompted 
initiatives to develop a stronger base of evidence about their impacts. These calls have been stimulated both by numerous 
system failures and research expressing concerns about the limitations of information systems methodologies in health 
care environments. Alongside the potential of information systems to produce positive, negative and unintended 
consequences, many measures of success, impact or benefit appear to have little to do with improvements in care, health 
outcomes or individual patient experiences. 

Combined these methodological concerns suggest the need for more detailed examination. This is particularly the case, 
given the prevalence within contemporary clinical and IS discourses on health interventions advocating the need to put the 
'patient at the centre' by engaging them in their own care and/or 'empowering' them through the use of information 
systems. 

This paper aims to contribute to these on-going debates by focusing on the socio-technical processes by which patients' 
interests and outcomes are measured, defined and evaluated within health interventions that involve them using web- 
based information systems. The paper outlines an integrated approach that aims to generate evidence about the impact of 
these types of health interventions that are meaningful at both individual patient and patient cohort levels. 
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Health care systems in Australia and across the world are 
facing major challenges in how to maintain the provision of 
quality care for all, in a manner that is economically 
sustainable. Ageing populations and the related growth in the 
incidence of chronic disease are contributing significantly to 
spiralling health costs. As a consequence system afford- 
ability has now become a major driver underpinning initia- 
tives aimed at health care reform. Currently, no health care 
system has developed a comprehensive response to meet 
these challenges but there is agreement and support for new 
models of service delivery that can enhance the efficiency 
and effectiveness of the system. 



INTRODUCTION 



what needs to be done and by whom, how, where and when. 
In this context, debates on health service delivery and quality 
of care have become increasingly prominent. Unfortunately, 
these debates have tended to be fragmented with a variety of 
approaches advocated each based on different measures and 
types of evidence to support their claims of value. It can 
however be argued that developing comprehensive solutions 
to the challenges faced requires a more integrated approach. 
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Considering the complexity of challenges faced, it is 
perhaps not surprising that there are different perspectives on 



"Most quality improvement activities in the 
world are still largely a reflection of the 
specific beliefs of specific parties about the 
best way to improve patient care. Crossing 
borders among professional pride and self- 
regulation, external accountability, payer 
profit, organisational development, and 
pleasing and involving patients can help us 
overcome the obstacles to optimal medical 
care" [1]. 



In parallel with these developments, the widespread 
adoption and utilisation of information systems in health care 
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has also been advocated as a major driver for stimulating the 
changes necessary to respond to the challenges faced. 

"Both Australian and international studies 
support investment in health information 
management and information & 
communication technologies (IM&ICT) as a 
fundamental component of system reform... to 
improve health outcomes while helping to 
offset the health cost increases expected as a 
result of advances in medical technologies and 
pharmacology, and the inevitable aging of the 
population. In addition, improving the 
collection, management and dissemination of 
health information will boost administrators' 
and policymakers' ability to undertake 
informed, system-wide planning, resulting in 
more effective and efficient deployment of 
health resources" [2]. 

By opening up opportunities for increased information 
access, improved information delivery, update and 
evaluation, the economic case for more information systems 
is strong. However, this deployment also raises a number of 
socio-technical, clinical and legal challenges that have 
become increasingly evident as the implementation of more 
sophisticated solutions have met with mixed success or 
failed to generate their forecast benefits [3]. This has led to 
calls for the utilisation of more sophisticated socio-technical 
approaches to the design, implementation and evaluation of 
health projects involving information systems [4, 5]. 

While debates on the role and impacts of different 
approaches are likely to continue, from a methodological 
perspective finding an integrated way to meaningfully 
evaluate the different types of evidence they use, may 
contribute to supporting more effective solutions to be 
developed. To explore these methodological issues, this 
paper considers how contemporary clinical and IS evaluation 
discourses on health interventions approach patients. More 
specifically, the paper focuses on the socio-technical 
processes by which patients' interests and outcomes are 
measured, defined and evaluated within health interventions 
that involve them using web-based information systems. The 
paper commences with a critical appraisal of the limitations 
of current clinical and IS evidence and evaluation processes 
of patients involved in these types of health interventions. 
The paper proceeds to outline an integrated approach that 
aims to generate evidence about the impact of these types of 
health interventions that are meaningful at both individual 
patient and patient cohort levels. 

CLINICAL EVIDENCE AND EVALUATION 

With the increasing incidence of chronic illness, health 
interventions encouraging people with a chronic disease to 
take a more active role in managing their own condition have 
become popular [6,7]. To date the evaluation of these 
interventions has focussed either on the impact of self- 
management upon health outcomes or on health resource 
usage and has been based on evidence produced through 
randomised controlled trials (RCTs). The evidence produced 
suggests that these health interventions can be as effective as 
the introduction of new medications [8]. However, there 
have been concerns raised about limitations of only 
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evaluating the impact of interventions involving patients 
through the quantitative techniques utilised in RCTs [9, 10]. 

The notion that all interventions within the medical field 
are comparable to a drug intervention and should therefore 
be evaluated in the same way, uncritically privileges 
evidence generated through randomised controlled trials over 
evidence generated in other ways [9, 10]. The persistent 
claims that RCTs are the 'gold standard' of health 
intervention evaluation results primarily because of the 
dominance of the scientific research paradigm within which 
healthcare exists [11]. RCTs merely reflect good 
experimental design, however there are many potential 
shortcomings in the design and implementation of RCTs [9, 
10]. Indeed the dominance of RCTs as 'the method' for 
evaluating health interventions has begun to attract some 
significant criticism [12] and the utility and validity of other 
approaches for use in evaluating health interventions have 
begun to emerge [13]. 

Major proponents of the RCT as the 'gold standard' for 
the evaluation of health interventions are advocates of 
evidence based medicine (EBM). EBM has the laudable goal 
of promoting the improvement of patient care through the 
use of up-to-date high quality, research evidence to make 
healthcare decisions. However, implicit within the EBM 
approach is a hierarchy of evidence that places evidence 
from RCTs and other population level research above other 
types of research evidence [10]. For health interventions, 
particularly those that engage the patient as an active 
participant, this hierarchy is of concern, as it prioritises 
population level evidence and reduces emphasis upon 
individuals. Related to this is the reality that health 
professional member organisations frequently base their 
development of clinical practice guidelines on the evidence 
from RCTs and systematic reviews. While not mandatory, 
these guidelines often place significant pressure upon 
individual health professionals to adhere to them, 
particularly for medico-legal reasons [14]. RCTs frequently 
concentrate on only a small number of measurable outcomes 
and may ignore patients' experiences and/or any unexpected 
effects from interventions that are more difficult to quantify 
or measure easily. During extended RCTs it is also difficult 
to isolate and evaluate any single intervention to avoid 
outside factors adversely affect the stability of the 
intervention [15]. 

More broadly, some researchers have raised questions 
about the dependence of RCTs upon their funding sources to 
produce outcome bias. This research was based upon a 
survey of published RCTs between 1990 and 2000 [16]. It 
revealed RCTs funded by for-profit organisations more 
frequently reported positive outcomes than those funded by 
not-for-profit organisations. These results raise questions 
around the validity of trial design and ethics. Perhaps most 
importantly they raise concerns in relation to the quality of 
the evidence that may now be used to treat patients. 

However, aligned to this point is the reality check that 
even where evidence is produced through RCTs, gaps 
continue to exist between this evidence and its actual use in 
clinical practice [17]. Others research have argued that where 
RCTs evidence is used there is a need for more effective 
mechanisms to tailor this evidence to individual patients 
[18]. Perhaps of more relevance here however, have been 
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critiques of how RCTs tend to marginalise individual patient 
experiences, priorities and perceptions [19, 20]. Most 
importantly however, the point needs to be made that whilst 
the goals of RCTs are laudable the practice, particularly 
where the interventions involve the patients themselves, 
continue to leave much to be desired. As has been identified: 

"Although evidence-based medicine advocates 
laudably took improving patient care as their 
goal, the methods they use to define and 
measure outcomes remained firmly in 
specialist hands. The collaborations set up by 
Cochrane and other similar groups were 
partnerships between researchers and 
clinicians, not doctors and patients... the most 
easily measured aspects of physiological 
change and therapeutic efficacy did not 
necessarily capture the markers of recovery 
that patients themselves deemed most 
important" [21]. 

It should be emphasized that the intention of this critical 
appraisal is not to present a unilateral argument for the 
discarding of RCTs but rather to highlight that there are 
some limitations that need to be considered. This is 
particularly the case where health interventions aim to 
actively involve patients and aim to generate evidence that 
meaningfully captures outcomes that reflect individual and 
cohort impacts [22]. The dominance of RCTs as a means of 
evaluating health care interventions continues to create a 
conflict between statistical evidence and the individual 
patient experience that needs to be more actively addressed if 
we are not to make the term patient-centred increasingly 
meaningless. 

Approaches to the Evaluation of Health Information 

Systems 

The diffusion of information and communication systems 
within the health system clearly has the potential to support a 
'better connected health system' that is more efficient, safer, 
knowledge-based, and patient-centred. However, there is a 
need for health information systems researchers to avoid 
downplaying the complexity and uncertainty that exists in 
the delivery of health care services or to make problematic 
assumptions about how information systems will benefit 
patients, health professionals and the health care system as a 
whole [23, 24]. 

Based on the history of health information systems 
deployments that have traditionally focused on health 
professionals and hospitals, it is not surprising that in 
developing evidence of their impact statistical methods 
including RCTs were until recently considered the methods 
of choice [23]. These evaluative approaches tried to compare 
the introduction or testing of an information system in a 
process analogous to the testing of a drug or other discrete 
intervention. Indeed, some health information systems 
researchers continue to be proponents of these methods [25]. 
However, the increasing penetration of the Internet and 
access and use of on-line health information by patients and 
citizens has made the use of other methods for evaluating the 
impact of information systems become more widely 
accepted [23, 26-29]. Similarly as the health information 
systems themselves have evolved the focus of evaluation has 
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moved from hardware and systems architectures towards the 
end-user where new techniques and evaluation criteria 
focused on usability, interfaces and content have been 
utilised [30-33]. This stated, there continues to be a need to 
improve the quality of reporting of evaluation studies [34]. 

More generally, there is a need to also recognise that 
reported benefits from the introduction of health information 
systems may be questionable or use measures of success that 
have little to do with improvements in care or patient 
outcomes [35, 36]. There is also a need for health 
information systems researchers to reflect on their 
underlying assumptions about the role, impact and 
importance of information. Particularly given that it is only 
one factor among many that influences health attitudes, 
perceptions, actions and outcomes. 

As health interventions involving patients' use of web- 
based information systems become more common, there are 
dangers for health information systems researchers of ending 
up treating patients as merely 'end-users' of their systems. 
When incorporating patients into design processes there is a 
need to continue to prioritise individual patient experiences, 
priorities and perceptions [24, 37, 38]. Indeed, supporting 
patients' active participation in their own care makes it 
imperative that we move beyond assessments of benefit in 
terms of the cohort so that systems engage with individual 
patient's and move us closer to genuine 'individualised care' 
and 'patient empowerment' [39]. 

In reviewing conventional approaches that have been 
used in the evaluation of health information systems it is 
clear that many have simply been adopted directly from 
techniques previously used in other industry sectors. For 
example, evaluations of systems development lifecycles 
have tended to be quantitative in nature and have almost 
exclusively focused on looking for intended benefits in terms 
of the technologies themselves, overall system performance 
or economic outcomes [26, 27]. There have also been large 
numbers of evaluation studies into interface design [40, 41], 
systems success factors [42, 43] and user satisfaction [44]. 

More recently, as the use of interpretivist research 
philosophies in information systems have become more 
widely accepted, user centred approaches to the evaluation of 
health information systems have become more common. 
These user-centred approaches have tended to focus on the 
continuous improvement of the system, with evaluation and 
review undertaken throughout the development lifecycle and 
with the input from end-users integral to the process [28]. 
Usability is a key concept within these approaches and refers 
to the degree to which an information systems is useful, 
efficient, effective and enjoyable to use [29]. However, 
within the health care domain these approaches acknowledge 
that information needs and flows can be difficult to 
determine precisely [28, 45]. 

There are now a vast array of methods employed within 
user-centred approaches including user-centred design 
(UCD), Human Computer Interaction (HCI) and 
Participatory Design (PD) [46]. Significantly, while this 
trend towards the utilisation of a variety of evaluation 
approaches is promising it brings with it a range of 
problems. It is now even more challenging to compare 
results generated from these approaches when considering 
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particular health information systems or applications [34]. 
Clearly the evaluation of different health information 
systems requires different methodologies. For example, an 
information system designed for a healthcare organisation 
and a collaborative work environment such as a hospital 
requires different methods in comparison to a system 
designed for an individual care provider or a healthcare 
consumer. This is even more important when considering 
how the socio-technical network of a system, its users, and 
the target organisation develops following its implementat- 
ion, and this itself requires further comprehensive study of 
the system's behaviour in context. Furthermore, both types 
of evaluation approaches, those used to investigate 
information systems within real implementation environ- 
ments and those used to evaluate information systems 
impacts at a single user level under controlled conditions, 
have limitations. For example, it can be argued that 
approaches that evaluate information systems under control- 
led conditions, such as simulation studies, may produce little 
insight into the impacts of these systems in a real-world 
organisation [47, 48]. 

Significantly, this brief review of approaches to the 
evaluation of health information systems suggests a division 
between health interventions that are primarily driven by 
technologists focused on improving 'information systems' 
and those that are primarily driven by health professionals 
focused on improving 'cohort health outcomes'. For both 
there is a need to genuinely consider the individual patient as 
participant in these interventions and what is implied when 
we talk about the 'patient at the centre', patient empower- 
ment and patient centred design [49]. 

KEEPING PATIENTS AT THE CENTRE 

Schneider and Lane [50] have argued: 

"Too often, a patient's treatments are 
performed "by the book" rather than being 
tailored to the patient's specific needs. To put 
it more bluntly, much of the care that today's 
patients receive is "medicine by numbers," 
which is best suited for that mythical 
abstraction, the "statistically average patient"". 

Health interventions involving patients using information 
systems present challenges to conventional clinical and 
information system approaches to the generation and 
evaluation of evidence on impacts and outcomes. 

From a clinical perspective the analysis of discrete sets of 
independent variables within RCTs is purposeful but tends to 
provide evidence of changes only at the cohort level of those 
involved in the health intervention. This is despite 
knowledge of the fact that the interplay of psychological and 
social factors directly impact upon an individual's self- 
perceived health status during their participation in clinical 
interventions [9-11, 15, 18, 19, 21]. 

From an information systems perspective user-centred 
approaches do involve the 'user' but may miss the patient! 
Focusing on technological acceptance or usability is sensible 
but it may often fail to examine in any detail the interplay of 
personal factors that make up an individual patients 
technology experiences, attitudes and responses. Further, in 
the realm of patients' as 'users' the implicit assumption of 
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mandatory use must be discarded as individual patients adopt 
and use systems in ways that suit themselves and their 
lifestyles and not necessarily as anticipated by clinicians or 
information systems professionals [23, 24, 26, 29, 31, 33, 45, 
49]. 

From a methodological perspective the tendency amongst 
these researchers to treat epistemology and method as being 
inseparable is evident [51, 52]. This is despite an awareness 
of the utility of employing a greater level of pragmatism 
when engaged in research with human participants [53]. 
Olson [54] argues that the distinction between qualitative 
and quantitative data is purely a distinction of data types and 
does not predict how these data should be treated and that 
research philosophy is of more importance in determining 
how data is analysed than the data type per se. Therefore in 
generating and evaluating evidence from health interventions 
involving patients use of web-based information systems, a 
case can be argued for analysing numeric and non-numeric 
data in an effort to 'keep the patient at the centre' [55]. 

Of course, working through the implications of these 
points is much more difficult in practice than in theory. From 
a patient's perspective the huge array of web-based health 
resources poses its own problems including differentials 
amongst individual patients to be able to categorise or 
recognise the validity and reliability of the content provided 
[33]. Although these resources may improve patient 
knowledge, their effect on objective clinical outcomes is 
inconsistent [56, 57]. It is also important to be realistic about 
the capacity of patients' to embrace web-based information 
systems, as financial and skill-based barriers may inhibit 
access to on-line information. These issues relate to broader 
concerns about how to maintain a patient focus in health 
interventions in order to avoid the creation of an 'eHealth 
divide' [49]. In the era of rapidly ageing populations and 
growing chronic illness it is also sobering to consider that it 
may continue to be those individuals with the most limited 
access to resources (financial, educational etc. and including 
Internet access) who are further marginalised by these types 
of health interventions. 

For clinical and information systems researchers these 
types of health interventions may also challenge some deeply 
held beliefs about 'what we think we are doing when we 
intervene' versus how this is perceived by our patients 
(participants). If researchers are really serious about 'keeping 
the patient at the centre' then the reality of discontinued use 
of these types of health interventions must be faced. 
Otherwise there is the risk of ending up trying to replace a 
dependency on health professionals for a dependency on 
information systems and the professionals who implement 
them [58]. There is a need to maintain an awareness of the 
fact that patients may be unwilling or unable to take on the 
new responsibilities of 'empowerment' and that even if they 
do, that this will axiomatically lead to improved quality of 
care whether evaluated at the individual or cohort level [49]. 
To date, evidence on the impact of attempts to 'empower' 
patients, and the actual impact this has on care practices and 
outcomes remains limited [59]. Indeed, despite the rhetoric, 
it appears evident that the impact of the availability of web- 
based clinical and other health information on patient-doctor 
relationships is mixed and dependent on a range of other 
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factors that are not related to information or the systems that 
make it accessible [21, 60]. 

METHODOLOGY 

Developing an Integrated Approach to Evaluating 
Evidence 

The issues and challenges for clinical and information 
systems researchers arising from health interventions 
involving patients using web-based information systems will 
continue to generate a range of approaches. From a 
methodological perspective, and with the aim of 'keeping the 
patient at the centre', this section of the paper outlines an 
integrated way to meaningfully evaluate different types of 
evidence on the impacts of these types of interventions. This 
approach recognises that a range of socio-technical processes 
impact directly on how patients' interests and outcomes are 
measured, defined and evaluated within health interventions. 
Critically, this approach argues that within the confines of a 
RCT involving patients use of web-based information 
systems, it is important to also examine individual 
participants. More specifically this involves exploring how 
individual's interpret and discuss their various experiences 
and to gain insight into individual experiences of the health 
intervention. It is argued that exploring and evaluating the 
interplay of factors that impact on individual patients' 
choices to adopt and subsequently use web-based 
information systems, and the effect these decisions and uses 
ultimately have on personal outcomes, perceptions and 
priorities is very important. 

Study Context 

Building on RCTs evaluation of intervention cohorts to 
investigate the experiences of individual patients is difficult 
and time-consuming but generates an increased depth of 
knowledge at both cohort and individual levels on the impact 
of a health intervention. The integrated approach briefly 
outlined here has already been employed within two 
controlled trials that aimed to assist patients develop self- 
efficacy for self-management of their chronic conditions and 
used web-based information systems [24, 37, 49, 59, 61]. 

In each controlled trial participants were introduced to a 
web-based information system as a non-mandatory part of 
the clinical intervention. Through the necessity to conform to 
the RCT protocols the information system was limited in 
scope. A subset from each RCT intervention group was 
selected. This subset consisted of two subgroups; patients 
who used the web-based system and patients who chose not 
to opting for a paper-based alternative. A three phase 
evaluation was undertaken to explore the differences in 
outcomes and experiences of the participants within each 
group. 

The first phase of the evaluation examined the project 
outcomes using the standardised outcome measures collected 
in the RCT in relation to the two sub-groups. These data 
were collected at baseline then quarterly for twelve months. 
Using these traditional outcome measures and comparative 
principles it could be assumed that any statistically 
significant differences between the two groups were the 
result of the influence of the web-based information system. 
The analysis undertaken used the mathematically derived 



standard measure, effect size, enabling comparison of change 
between or within groups [62, 63]. 

The second phase of the investigation used audio- 
recorded semi-structured interviews to explore the 
experiences of the participants from these sub-groups to 
examine how the use of the information system had 
influenced their experiences within the RCT. Interviews 
were transcribed and then thematic coding was used to 
analyse the data [64]. The coded interview data was 
compared across and within the two sub-groups to establish 
similarities and differences between and within the groups. 

The third phase involved the combination of data from 
phases one and two for each of the participants, and explored 
in detail the inter-relationships amongst factors influencing 
participant's health outcomes and experiences over the 
duration of their participation in the trial. Phase three 
provided access to a rich and highly detailed picture of the 
individual participant's health outcomes and experiences and 
allowed the research to expand the depth of their 
investigation and understanding. This process can be seen to 
reconstruct the individual and their experiences and enabled 
the research to develop individual case studies that provided 
a range of evidence to expand upon the data analysis from 
the previous two phases [61]. 

SUMMARY 

Using this methodology in both of the health 
interventions led to the generation of some very interesting 
results. In both instances phase one analysis based on 
conventional statistical analysis of the cohort clearly and 
unequivocally revealed that the web-based information 
system did not have a positive effect and there was the 
possibility of some minor negative effect. This evidence was 
common across all statistical measures deployed. However, 
analyses generated from phases two and three revealed a 
more complex account of the intervention and the impacts on 
user experiences, capacities for behavioural change, and the 
influence of individual disease knowledge and stage. 

More specifically, it was highlighted that at the indivi- 
dual level there were a highly varied set of reactions to the 
introduction of the web-based information systems that were 
not revealed in phase one analysis. These reactions result 
from the inter-relationships between many individual factors 
including the severity of disease, self-image, family context, 
confidence, social support and disease understanding. 

The results demonstrated many different experiences and 
unintended consequences including a participant with severe 
disease, who lived alone, who reported a transformation of 
their life through the use of solitaire, email and changed 
family status. For this individual patient, this did lead to an 
upward trend in symptoms for some time even though this 
trend was not statistically significant. This participant also 
continued to use the web-based information system for the 
duration of the project and found some benefit from this in 
terms of managing their disease and also considered that it 
made a difference to them in terms of quality of life. Another 
participant, who initially had elected to adopt the web-based 
information system, did not manage to use the system or the 
computer at all during the intervention. This participant had 
a moderately severe chronic condition and was primary carer 
for her husband who had a similar but severe chronic 
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condition. Her husband was extremely unsupportive of her 
participation in the health intervention and refused to allow 
her access to further training in the use of the web-based 
information system. Despite these obstacles this participant 
persevered and acquired a good understanding of the aims of 
the health intervention and improved her self-efficacy for 
self-management through her participation [61]. 

CONCLUSIONS 

This paper has considered how contemporary clinical and 
IS evaluation discourses on health interventions approach 
patients. More specifically, the paper has focused on the 
socio-technical processes by which patients' interests and 
outcomes are measured, defined and evaluated within health 
interventions that involve them using web-based information 
systems. 

The paper has presented a critical appraisal of the 
limitations of current clinical and IS evidence and evaluation 
processes in relation to patients involved in these types of 
health interventions. It has also outlined an integrated 
approach to evaluation that has been used to generate 
evidence about the impact of these types of health 
interventions. This approach has produced evidence about 
the impact of the intervention and the web-based information 
systems that are meaningful at both individual patient and 
patient cohort levels. The integrated approach outlined 
encourages the utilisation of both quantitative and qualitative 
data across cohorts, groups and individuals to build a 
detailed view of participants' experiences and outcomes 
within this type of health intervention. By combining the 
analysis of RCT data with interview data this approach 
allows for evidence to be generated at multiple levels of 
abstraction that more readily supports 'keeping the patient at 
the centre' in evaluation. 

As the number of health interventions directly involving 
patients and their use of information systems increase it is 
anticipated that the approach outlined will contribute to more 
comprehensive evaluations of the impact of interventions on 
patients and to the accommodation of perspectives that 
patients themselves recognise, acknowledge and deem 
relevant. 
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